{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RL agent in the Grid World \n",
    "\n",
    "Let's strengthen our understanding of RL by looking at another simple example.\n",
    "Consider the following grid world environment:\n",
    "\n",
    "![title](Images/4.png)\n",
    "\n",
    "The positions A to I in the environment are called the states of the environment.\n",
    "The goal of the agent is to reach state I by starting from state A without visiting\n",
    "the shaded states (B, C, G, and H). Thus, in order to achieve the goal, whenever\n",
    "our agent visits a shaded state, we will give a negative reward (say -1) and when it\n",
    "visits an unshaded state, we will give a positive reward (say +1). The actions in the\n",
    "environment are moving up, down, right and left. The agent can perform any of these\n",
    "four actions to reach state I from state A.\n",
    "\n",
    "The first time the agent interacts with the environment (the first iteration), the agent\n",
    "is unlikely to perform the correct action in each state, and thus it receives a negative\n",
    "reward. That is, in the first iteration, the agent performs a random action in each\n",
    "state, and this may lead the agent to receive a negative reward. But over a series of\n",
    "iterations, the agent learns to perform the correct action in each state through the\n",
    "reward it obtains, helping it achieve the goal. Let us explore this in detail.\n",
    "\n",
    "## Iteration 1:\n",
    "\n",
    "As we learned, in the first iteration, the agent performs a random action in each state.\n",
    "For instance, look at the following figure. In the first iteration, the agent moves right\n",
    "from state A and reaches the new state B. But since B is the shaded state, the agent\n",
    "will receive a negative reward and so the agent will understand that moving right is\n",
    "not a good action in state A. When it visits state A next time, it will try out a different\n",
    "action instead of moving right:\n",
    "\n",
    "![title](Images/5.PNG)\n",
    "\n",
    "As the avove figure shows, from state B, the agent moves down and reaches the new state\n",
    "E. Since E is an unshaded state, the agent will receive a positive reward, so the agent\n",
    "will understand that moving down from state B is a good action.\n",
    "\n",
    "From state E, the agent moves right and reaches state F. Since F is an unshaded state,\n",
    "the agent receives a positive reward, and it will understand that moving right from\n",
    "state E is a good action. From state F, the agent moves down and reaches the goal\n",
    "state I and receives a positive reward, so the agent will understand that moving\n",
    "down from state F is a good action.\n",
    "\n",
    "\n",
    "## Iteration 2:\n",
    "\n",
    "In the second iteration, from state A, instead of moving right, the agent tries out a\n",
    "different action as the agent learned in the previous iteration that moving right is not\n",
    "a good action in state A.\n",
    "\n",
    "Thus, as the following figure shows, in this iteration the agent moves down from state A and\n",
    "reaches state D. Since D is an unshaded state, the agent receives a positive reward\n",
    "and now the agent will understand that moving down is a good action in state A:\n",
    "\n",
    "\n",
    "![title](Images/6.PNG)\n",
    "\n",
    "As shown in the preceding figure, from state D, the agent moves down and reaches\n",
    "state G. But since G is a shaded state, the agent will receive a negative reward and\n",
    "so the agent will understand that moving down is not a good action in state D, and\n",
    "when it visits state D next time, it will try out a different action instead of moving\n",
    "down.\n",
    "\n",
    "From G, the agent moves right and reaches state H. Since H is a shaded state, it will\n",
    "receive a negative reward and understand that moving right is not a good action in\n",
    "state G.\n",
    "\n",
    "From H it moves right and reaches the goal state I and receives a positive reward, so\n",
    "the agent will understand that moving right from state H is a good action.\n",
    "\n",
    "\n",
    "## Iteration 3:\n",
    "\n",
    "In the third iteration, the agent moves down from state A since, in the second\n",
    "iteration, our agent learned that moving down is a good action in state A. So, the\n",
    "agent moves down from state A and reaches the next state, D, as the following figure shows:\n",
    "\n",
    "![title](Images/7.PNG)\n",
    "\n",
    "Now, from state D, the agent tries a different action instead of moving down since in\n",
    "the second iteration our agent learned that moving down is not a good action in state\n",
    "D. So, in this iteration, the agent moves right from state D and reaches state E.\n",
    "\n",
    "From state E, the agent moves right as the agent already learned in the first iteration\n",
    "that moving right from state E is a good action and reaches state F.\n",
    "\n",
    "Now, from state F, the agent moves down since the agent learned in the first iteration\n",
    "that moving down is a good action in state F, and reaches the goal state I.\n",
    "\n",
    "The following figure shows the result of the third iteration:\n",
    "![title](Images/7.PNG)\n",
    "\n",
    "As we can see, our agent has successfully learned to reach the goal state I from state\n",
    "A without visiting the shaded states based on the rewards.\n",
    "\n",
    "In this way, the agent will try out different actions in each state and understand\n",
    "whether an action is good or bad based on the reward it obtains. The goal of the\n",
    "agent is to maximize rewards. So, the agent will always try to perform good actions\n",
    "that give a positive reward, and when the agent performs good actions in each state,\n",
    "then it ultimately leads the agent to achieve the goal.\n",
    "\n",
    "Note that these iterations are called episodes in RL terminology. We will learn more\n",
    "about episodes later in the chapter."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
